An assessment of submissions made to the Predictive Toxicology Evaluation Challenge

Constructing "good" models for chemical carcinogenesis was identified in IJCAI-97 as providing a substantial challenge to "knowledge discovery" programs. Attent ion was drawn to a comparative exercise which called for predictions on the outcome of 30 rodent carcinogenicity bioassays. This the Predictive Toxicology Evaluation (or PTE) Challenge was seen to provide AI programs wi th an opportunity to participate in an enterprise of scientific merit , and a yardstick for comparison against strong competit ion. Here we provide an assessment of the machine learning (ML) submissions made. Models submitted are assessed on: (1) their accuracy, in comparison to models developed wi th expert collaboration; and (2) their explanatory value for toxicology. The principal findings were: (a) using structural informat ion available from a standard modelling package, layman-devised features, and outcomes of established biological tests, results from MLderived models were at least as good as those wi th expert-derived techniques. This was surprising; (b) the combined use of structural and biological features by ML-derived models was unusual, and suggested new avenues for toxicology modelling. This was also unexpected; and (c) significant effort was required to interpret the output of even the most "symbolic" of ML-derived models. Much of this could have been alleviated wi th measures for converting the results into a more "toxicology-friendly" form. As it stands, their absence is sufficient to prevent a whole-hearted acceptance of these promising methods by toxicologists. This suggests that ML techniques have been able to respond not fully, but nevertheless substantially to the P T E Challenge. 1 I n t r o d u c t i o n In his essay "Two conceptions of science" [Medawar, 1984], the distinguished biologist Peter Medawar describes the valuation of contributions to science thus:Here then are some of the criteria used by scientists when judging their colleagues' discoveries and the interpretations put upon thern. Foremost is their explanatory value their rank in the grand hierarchy of explanations and their power to establish new pedigrees of research and reasoning. A second is their clarifying power, the degree to which they resolve what has hitherto been perplexing. .. Explanations that reach this stage of inspection are usually understood to have achieved an acceptable level of accuracy, however measured. W i th the emergence of ML programs capable of constructing empirical generalisations from scientific data, it is possible to examine the extent to which such machine-authored descriptions meet the criteria used to judge their human counterparts. This is of special interest if such programs are intended to act as genuine scientific assistants to experts. One area for conducting such an examination was proposed in the form of the Predictive Toxicology Evaluation Challenge (in IJCAI-97, see [Srinivasan et a/., 1997]). The problem of predicting chemical carcinogenesis described there is part icularly well-suited as a testbed for a number of reasons. Besides its undisputed humanitarian value, principal reasons are that: (1) there is an urgent need for low-cost, accurate toxicity models that can reduce a reliance on slow, expensive rodent bioassays [Bristol et a/., 1996]; (2) there is much to be learnt about the molecular mechanisms underlying carcinogenic activi ty; and (3) there is a well-established scientific programme wi th in the U.S. National Toxicology Program (NTP) concerned with the comparative evaluation of toxicity models (which may be of human origin, see: dir.niehs.nih.gov/dirlecm/pte2.htm1). These provide machine-based "hypothesis constructors" the opportuni ty to construct accurate models, which may yield 1 Al l Internet sites mentioned in this paper are to be prefixed with http:// unless otherwise indicated 270 CHALLENGE PAPERS new insights and subject to review much in the manner described by Medawar. This paper reports on the machine learning (ML) submissions made to this I J C A I challenge from its inception in August, 1997 to December, 1998. The paper is organised as follows. Section 2 summarises the course of the challenge from 1997, and presents the models selected for further evaluation. Section 3 contains an assessment of the accuracies of the ML models in comparison to those developed under the guidance of expert toxicologists (this includes toxicology expert systems). Section 4 contains an appraisal of the explanatory value of the ML models. Section 5 concludes this paper. 2 T h e I J C A I P T E Chal lenge: detai ls